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Abstract 

A paper appearing in this journal by Klein, Hamilton, McCaffrey and Stecher 
(2000) attempted to raise serious questions about the validity of the gains in 
student performance as measured by Texas’ standardized test, the Texas 
Assessment of Academic Skills (TAAS). Part of their analysis was based on the 
results of three tests which they administered to 2,000 fifth grade students in 20 
Texas schools. Although Klein et al. indicated that the 20 schools were not selected 
in a way which would insure that they were representative of the nearly 3,000 Texas 
schools that enrolled fifth graders, generalizations based upon the results for those 
schools were nonetheless offered. The purpose of this short paper is to 
demonstrate just how unrepresentative the 20 schools used by Klein et al. actually 
were, and in so doing to cast doubt on certain of their conclusions. 


1 The author wishes to thank Prof. Jon Lorence for numerous constructive suggestions. Accepted under the 
editorship of Sherman Dorn. Send commentary to Casey Cobb (casey.cobb@uconn.edu). 
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This is a reply to the paper “What do Test Scores in Texas Tell Us?”, by Stephen P. Klein, 
Laura S. Hamilton, Daniel F. McCaffrey, and Brian M. Stecher (2000). 

Klein et al. (2000) presented numerous findings questioning the effectiveness of the Texas 
accountability system. They specifically challenged the validity of the Texas Assessment of Academic 
Skills (TAAS) test, required of all eligible students in the Texas public school system. Their 
conclusions were based on several sets of analyses. First they estimated the association between 
student test scores and student economic status. Other analyses examined the relationships between 
student performance on the TAAS and three different tests which they administered in the spring of 
1997 to some 2,000 fifth graders in 20 schools, described only as “a mix of 20 urban and suburban 
schools in Texas.” Although results based on individual students were as anticipated, aggregate 
school-level findings indicated that high TAAS performance was largely unrelated to scores on 
standardized tests not used in the state accountability system. In addition, Klein et al. did not find 
the expected negative linear association between the percentages of disadvantaged children and test 
scores commonly reported in the research literature. Although Klein et al. suggested that the 
discrepancies “could be due to the unique characteristics of the 20 schools in our study or other 
factors,” they nonetheless concluded that their findings indicated that observed increases in Texas 
student academic performance reported by Grissmer et al. (2000) were highly suspect. However, 
analyses based on all Texas public schools reported in this note reveals that part of the Klein et al. 
findings were incorrect because the campuses selected for their study were not representative of the 
population of elementary schools. 



Proportion of Students Receiving Subsidized Lunches 


Figure 1. Campus Mean Math Scores Versus Proportion of 
Disadvantaged Students, Texas Fifth Graders, 1997 TAAS Test 


Figure 1 (above) shows the relationship between student TAAS test scores and economic 
disadvantage based on Texas public schools which tested fifth-grade students in mathematics in the 
spring of 1997. To be consistent with Klein et al., only 1997 fifth grade TAAS mathematics scores 
are utilized. The results are shown for non-special education students whose test results were 
included in the state’s accountability rating system. The scatterplot in Figure 1 is based on 225,433 
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students attending 2,825 campuses. A few schools which tested fewer than 20 students were 
excluded from the analysis in order to obtain reliable estimates of average school performance. The 
test performance measure, shown on the vertical axis, is the simple average for each campus of the 
number of questions answered correctly. The horizontal axis represents the proportion of 
economically disadvantaged tested students as measured by enrollment in the federally-subsidized 
free or reduced-price lunch programs. 



Percentage ol students in Vie lunch program 

Figure 5 

Relationship Between Mean TAAS Math Score 

and Percentage of Students in the Lunch Program 

When the School Is Used As the Unit of Analysis 

Figure 2. Diagram appeared as Figure 5. 

In Klein, et al. (2000) 

Although campuses testing fewer than 20 students are omitted, the picture is intended to 
show the universe from which the twenty campuses used in the study by Klein, et al, were drawn. 
The corresponding scatterplot based on the 20 campuses analyzed by Klein et al. is reproduced in 
Figure 2 (above). When the figure from the Klein study is compared with the figure based on all 
Texas schools, it is apparent how unrepresentative those 20 schools were with respect to 
characterizing the relationship between performance on the fifth grade math TAAS test and the 
average proportion of students who were economically disadvantaged. Consequently, the Pearson 
correlation between average math performance and economic disadvantage based on the 2,825 
campuses differs considerably from the correlation Klein et al. reported. Whereas their correlation 
was only 0.13, and positive, the correlation from the more comprehensive set of campuses shown is 
a -0.55. This latter figure has the expected large negative value, and is comparable to the correlation 
coefficients which they found between SES and the non-TAAS tests which they administered. 
Those correlations ranged between -0.66 to -0.76. Although there is a degree of heteroscedasticity 
in the figure based on the more representative set of schools, due perhaps to ceiling-effect in the 
low-poverty schools, there is no indication of the u-shaped association evident in Figure 2, based on 
the Klein et al. data. 

In sum, the results derived from the tests Stecher and Klein administered in 1997 to 2,000 
students in 20 schools are highly unrepresentative of Texas elementary schools and should not be 
used to draw conclusions about the relationship between poverty and school performance in Texas. 
The analysis presented here supports the validity of the TAAS test. Additional, more 
comprehensive, school-level analyses examining the association between performance on the TAAS 
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and other standardized tests are required to ascertain whether the Klein et al. cross-test findings 
based on 20 elementary campuses can be generalized. 

There is no intent here to suggest that the Texas accountability system is immune from 
criticism. Analyses of NAEP data have also raised questions pertaining to the validity of learning 
gains reported on Texas. To illustrate, even RAND Corporation researchers differ among 
themselves in their interpretation of the meaning of NAEP test results for Texas. Whereas Klein et 
al. (2000) contended NAEP findings showed the Texas accountability system is ineffective, 

Grissmer et al. (2000) suggested that the implementation of stricter state educational requirements in 
Texas helped improve student academic learning. The current note does not address this issue of 
validity (but see Toenjes and Dworkin 2002; Toenjes, Dworkin, Lorence and Hill 2002). My findings 
are more limited, but clearly demonstrate that the absence of a negative relationship between school- 
level test performance and economic disadvantage reported by Klein et al., based on 20 
unrepresentative schools, does not support their claim that the TAAS is an invalid measure of 
student achievement. 
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